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We investigate factors influencing student sign-ups for Washington State's College Bound Scholarship program and consider 
whether there is scope for the program to change college enrollment expectations. We find that student characteristics asso- 
ciated with signing the scholarship pledge closely parallel characteristics of low-income students who attend 4-year colleges, 
suggesting that signing the pledge is driven largely by preexisting expectations of college going. We also find evidence that 
student sign-up rates are lower than have been previously reported, which is important given the perception among program 


administrators that nearly all eligible students sign up. 
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CoLLeGE and credential completion rates have recently 
increased for the first time since the Great Recession 
(Shapiro et al., 2016, 2017a), but college attainment gaps 
among various demographic and socioeconomic groups 
persist, with minority and low-income students being far 
less likely to obtain college degrees (Shapiro et al., 2017a, 
2017b; U.S. Department of Education, 2016). One factor 
that may contribute to this gap is low expectations of col- 
lege attendance due to the unaffordability of college for dis- 
advantaged youths. For instance, Perna (2008) finds that 
perceptions of college affordability are likely to matter to 
young people and that perceptions about college attainabil- 
ity help shape their high school trajectories (Hill & Wang, 
2015; Howard, 2003). 

A state policy response to the relatively low level of col- 
lege attainment of disadvantaged students is to make college 
more affordable and also alert disadvantaged students early, 
letting them know that college is within their grasp. 
One such program is an early-commitment, need-based 
scholarship in which states or local areas commit early in a 
low-income student’s schooling to provide financial aid for 


college financial aid, low-income youth, scholarship programs, program take-up 


college if, in return, the student pledges to do well in school 
and stay out of trouble. A central theory of action of these 
early-commitment programs is that the pledge will change 
students’ expectations about college while they still have 
time to prepare for college. 

There is very little evidence about the factors that predict 
whether students sign up for these early-commitment pledge 
programs.' This is an important gap in the financial-aid and 
college-going literature given that students can financially 
and academically benefit from such programs only if they 
sign up for them. 

This paper evaluates Washington State’s College Bound 
Scholarship (CBS), an early-commitment scholarship that 
promises full tuition to low-income middle school students 
who participate. By examining student-level historical data 
and conducting qualitative interviews, this paper addresses 
the following questions: (a) What student and middle school 
characteristics are associated with signing the CBS pledge? 
(b) How closely do these characteristics parallel preprogram 
predictors of college enrollment? (c) What do program 
administrators report doing to encourage student uptake of 
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the program? We address the first two questions through 
quantitative analyses, using administrative data from 
Washington State. The third question is addressed by a qual- 
itative component of our work that involves semistructured 
interviews with administrators from the College Success 
Foundation (which, as we describe later, is involved with 
outreach efforts to promote the CBS) and middle school 
guidance counselors. 

We find that factors associated with signing up are similar 
to factors associated with enrollment in college, and this pat- 
tern holds for each of the first three cohorts of CBS-eligible 
students. Indeed, there is little evidence that the demographic 
patterns predicting sign-ups change over the first three 
cohorts of program administration, calling into question the 
extent to which the pledge program is changing college- 
going expectations. Additionally, we find there is a positive 
correlation between predicted and actual sign-up rates at the 
school level, but the correlation (.40) is not overwhelmingly 
high, suggesting that targeting resources toward encourag- 
ing student sign-ups at schools with surprisingly low sign-up 
rates could significantly boost overall program enrollment. 


Conceptual Model/Theory of Action for Early- 
Commitment Pledge Programs 


Human capital theory posits that individuals make deci- 
sions about human capital investments that are thought to 
enhance productivity and earnings (Altonji, Blom, & 
Meghir, 2012; Becker, 1993). Early-commitment pledge 
programs may change the calculus when it comes to deci- 
sions about investing in college preparation by providing an 
early signal that the financial resources to attend college are 
within a student’s grasp. 

However, the extent to which pledge programs influence 
college going is also likely to be influenced by the informa- 
tion that students receive about programs as well as the guid- 
ance they get about how to prepare for college success 
(Oreopolous & Dunn, 2012). The role played by school 
administrators in providing students information about col- 
lege is likely to be particularly important for low-income 
students given that research (Avery & Kane, 2004) finds that 
they are substantially less likely to have an accurate picture 
of college admissions processes or costs. 

The CBS provides clear guidance about academic and 
behavioral requirements for the receipt of a scholarship, 
along with the promise that students adhering to the require- 
ments are likely to have the resources necessary to attend 
college (assuming they remain income eligible in 12th 
grade); for most eligible students, the program essentially 
eliminates the cost of attending a Washington public college 
(aside from room and board). Thus, the CBS should increase 
forward-looking students’ expectations about college (and 
hence make them more likely to prepare for college in high 
school, apply to colleges, and enroll). 


Students must sign up in middle school for the CBS in 
order to be eligible for the scholarship in the 12th grade (more 
details about the CBS are available in Appendix A online). 
We believe signing up is an early indicator of college-going 
expectations. But students eligible for the CBS will also 
have heterogeneous expectations about college: Some, for 
instance, are likely to go to college irrespective of the CBS 
program. The hope is that the program, as it is rolled out over 
time, is encouraging eligible students who had not previously 
considered college to make college an expectation. Thus, 
given the theory of action, we test two hypotheses: 


Hypothesis 1: The factors that predict college going 
among eligible students will also predict the likeli- 
hood of signing up for the CBS program. 

Hypothesis 2: As the program is rolled out over time, the 
relationship between the factors predicting college 
going and signing up for the program will be lessened. 


The first hypothesis reflects the fact that students with a 
high propensity to go to college should also have a high pro- 
pensity to sign up for a program that provides them with a 
college scholarship. The second hypothesis reflects the hope 
that as the program is implemented and word spreads about 
the availability of this source of college funding, students 
who would not previously have considered college as an 
option begin to do so, weakening the relationship between 
the factors predicting college and program sign up. 


Literature Review 


Several states have implemented financial aid programs 
designed to address college enrollment and attainment gaps 
by offering in-state low-income adolescents an early prom- 
ise of funding for college in exchange for a student’s early- 
commitment pledge (typically during seventh to ninth 
grades). Pledges commonly ask students to do well in high 
school, be a good citizen, and apply to college. Currently 
there are three states (Indiana, Oklahoma, and Washington) 
that are operating early-commitment pledge programs that 
meet Blanco’s (2005) “three core criteria for early commit- 
ment programs: that they make a guarantee of aid; that aid is 
designated only for economically disadvantaged students; 
and that students are identified in elementary, middle school, 
or early high school” (p. 9).” 

These early-commitment pledge programs are similar to 
other promise programs that spread rapidly in recent decades 
(Dynarski, 2004; LeGower & Walsh, 2017).° The effects of 
these promise programs, that is, programs in which states or 
local municipalities make a commitment to fund at least 
some part of in-state college tuition, have been studied 
extensively. The findings tend to show positive effects for 
in-state college matriculation (with some diversion of stu- 
dents from out of state to in state) and credit accumulation 


(e.g., Bartik, Hershbein, & Lachowka, 2017; Carruthers & 
Ozek, 2016; Cornwell, Mustard, & Sridhar, 2006; Page, Iriti, 
Lowry, & Anthony, in press; Perna & Leigh, 2017; Scott- 
Clayton, 2011; Sjoquist & Winters, 2014). Yet, these pro- 
grams may also exacerbate college attainment gaps between 
advantaged and disadvantaged students. For instance, 
Dynarski (2000) finds that “Georgia’s program has likely 
increased the college attendance rate of all 18- to 19-year- 
olds by 7.0 to 7.9 percentage points” yet “widened the gap in 
college attendance between blacks and whites and between 
those from low- and high-income families” (p. 629).* 

Although early-commitment pledge programs are similar 
to other promise programs insofar as they typically offer aid 
at in-state colleges for students achieving modest academic 
thresholds, they differ from other promise programs in a key 
way: They require students to sign an early-commitment 
pledge as a condition of eligibility. This programmatic ele- 
ment is designed to increase college going and, importantly, 
close college attendance gaps. In particular, they are thought 
to help low-income students directly by making college 
more affordable and indirectly by signaling to them early 
enough that college is financially within their reach. In doing 
so, the early promise of a college scholarship is expected to 
raise students’ expectations about the feasibility of college 
attendance and create a strong incentive for them to do well 
in high school and fulfill pledge requirements. 

It is worth emphasizing that the opt-in nature of the 
pledge requirement is a feature designed to target students, 
particularly those who, due to financial concerns, might not 
have college-going expectations entering high school. 
College-going aspirations are expected to change high 
school behavior, for example, to lead students to take high 
school courses that ready them for college. But if opting in 
has no effect on subsequent behavior in high school, then it 
is likely that early-commitment pledge programs will be less 
effective than other promise programs wherein eligibility is 
automatic, because some students will fail to make them- 
selves eligible for scholarships by not signing up; Thaler and 
Sunstein (2003) show in a variety of contexts that programs 
requiring participants to opt in tend to have lower participa- 
tion rates than programs in which participation is the default 
option. 

There is a striking dichotomy between the aspirations of 
younger students to attend college and the actuality of college 
enrollments.” And when college funding is available, low- 
income students are less likely to know about it, are less 
familiar with the process for attaining it, and/or lack a good 
understanding of the academic requirements needed to be 
admitted and to succeed in college (Bettinger, Long, 
Oreopoulos, & Sanbonmatsu, 2012; Dynarski & Scott- 
Clayton, 2006; Hahn & Price, 2008; Kirst, Venezia, & 
Antonio, 2004; B. Long & Riley, 2007; Rosenbaum, 2001). 
Not surprisingly, then, by senior year, low-income students 
are substantially less likely to have taken courses necessary 


Sign Me Up 


to succeed in college (Choy, 2001; Jacob & Linkow, 2011). 
Nevertheless, Castleman and Goodman (2018) found that 
intensive high school counseling benefits low-income stu- 
dents, and the program they studied “successfully shifts 
enrollment toward four-year colleges,” which further resulted 
in “persistence through the third year of college” (p. 19). 

Existing studies on early-commitment opt-in pledge pro- 
grams are limited in their ability to identify treatment effects 
as they lack data needed to form appropriate comparison 
groups for those students who are eligible to participate in 
these programs. St. John and colleagues (St. John et al., 
2004; St. John, Fisher, Lee, Daun-Barnett, & Williams, 
2008; St. John, Gross, Musoba, & Chung, 2005; St. John, 
Hu, & Weber, 2001; St. John, Musoba, & Simmons, 2003) 
investigated the impact of Indiana’s program yet lacked both 
data on cohorts of students before the introduction of the 
pledge program and data needed to identify if a student was 
eligible for the program. As a result, they were forced to 
compare students who signed the pledge with a comparison 
group of students who may or may not have been eligible.° 
Fortunately, as we describe later, we have access to both 
CBS eligibility indicators and data on prepolicy cohorts of 
students. This allows us to explore the extent to which fac- 
tors predicting whether students sign the pledge look similar 
to the factors associated with college enrollment from before 
the implementation of the CBS. In other words, we are able 
to investigate the degree to which the aspirations surround- 
ing college attendance of different groups of students do or 
do not change.’ 


Methods and Data 
Quantitative Analytic Methods 


We begin with a quantitative analysis of the individual 
and school characteristics that predict the likelihood of a stu- 
dent signing the CBS pledge in middle school. We estimate 
the specifications shown in Equations la and 1b using data 
on the first three cohorts of CBS-eligible students: 


Signed ;, = F (Bo + BX; + MSCE,, + Ein ). (Ja) 


Signed = F (Bo + BX; + BoSiy, + Eim ). (1b) 


The subscript i denotes the student and m denotes the 
middle school attended in eighth grade.* The dependent vari- 
able for this analysis, Signed,,,, is an indicator variable that 
equals | if the student signed the pledge by the end of eighth 
grade (or ninth grade for Cohort 1°). The student-level pre- 
dictors (i.e., the X, vector) include standardized scores on 
Washington’s reading and mathematics assessments— 
known as the Washington Assessment of Student Learning 
(WASL)—when the student was in sixth grade and indica- 
tors for taking the WASL tests out of grade level and taking 
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a modified version of the WASL,’° student’s age in May of 
eighth grade, and indicators for female, race-ethnicity group 
(Hispanic; non-Hispanic African American, Asian, Hawaiian 
or Pacific Islander, or Native American or Alaskan Native; 
or non-Hispanic and two or more races), disability status, 
migrant status, homeless status, “highly capable” status, 
“transitional bilingual” status, language spoken at home 
other than English, and attendance at a public school part- 
time (as a homeschooler or private school enrollee). 

We estimate Equation la using a logit specification with 
middle school fixed effects, MSCE,,, to account for time- 
invariant school-level factors that may influence students’ 
signing the pledge.'! We cluster the standard errors of all the 
regressions at the middle school level to account for the fact 
that guidance counselors and school culture are both thought 
to be important determinants for whether middle school stu- 
dents sign the pledge; that is, there is likely to be within- 
cluster error correlation (Cameron & Miller, 2015). 

We estimate Equation 1b (which drops middle school 
fixed effects and adds middle school characteristics, S,,,) 
using a hierarchical logistic regression (i.e., the logit analog 
of hierarchical linear modeling) with random intercepts. The 
advantage of the fixed-effects specification (Equation la) is 
that it accounts for both observable and unobserved time- 
invariant school-level factors. The disadvantage of Equation 
la over Equation 1b is that the former does not estimate the 
way that student sign-up rates are influenced by observable 
school-level characteristics. The following middle school 
characteristics are included in Equation 1b: percentage of 
students receiving free or reduced-price lunch (FRPL), mean 
standardized student score on the sixth-grade mathematics 
WASL among eighth-grade students in the school, eighth- 
grade enrollment in fall (divided by 100), whether the school 
has a guidance counselor, the proximity to college slots 
(measured as the number of undergraduate students at a 
4-year college within a 50-mile radius’), and an indicator 
for the school’s region (Puget Sound [including King, Pierce, 
Kitsap, Thurston, and Snohomish counties], the remainder 
of western Washington [i.e., west of the Cascade Mountains], 
or eastern Washington). After running these logistic regres- 
sions, we compute the marginal effect of a one-unit increase 
in each variable (holding all other variables constant) on the 
probability that the student signs the pledge. 

Next, we estimate versions of Equations la and 1b where 
we replace the outcome with an indicator for enrolling in any 
college within 4 years of ninth-grade entry, Enrolled,,.'° 


im 


Enrolled im = F (Yo + 1X; + MSCE », + Em )- (2a) 


Enrolled jm =F (yo +91 X5 +725 + Ein) (2b) 


For these regressions, we use data from two pre-CBS 
cohorts.'* These regressions allow us to evaluate whether 


there are meaningful differences between the B, and B, vec- 
tors of coefficients using “Signed” as the dependent variable 
and the y, and y, vectors of coefficients using “Enrolled” as 
the dependent variable. If these sets of coefficients are simi- 
lar, it would indicate that the types of students who are more 
prone to attend college are also more likely to sign the 
pledge. Substantial differences in the coefficients may also 
provide useful information.'° 

To assess the similarity of the estimated vectors of coef- 
ficients using “Enrolled” rather than “Signed” as the depen- 
dent variable, we examine the correlation between the sets of 
marginal effects found for the two dependent variables. 
Then, to gain a better sense of whether the CBS program is 
encouraging students who are demographically unlikely to 
attend college, we compare the predicted probabilities gen- 
erated by the enrollment and pledge models. We investigate 
whether the predicted enrollment and pledge probabilities 
align or diverge for each cohort of students eligible to sign 
the CBS pledge. Specifically, we assess the extent to which 
the estimated coefficients from specification 2b (the coeffi- 
cients from the college enrollment model) predict the esti- 
mated probabilities of signing the pledge for each 
CBS-eligible cohort (generated from estimating specifica- 
tion 1b, the pledge model, separately by cohort).'° 

More formally, we generate two predicted probabilities 
for each individual in each  postpolicy cohort: 
Pricm = F Boe +6, f +058. , which is the predicted 
probability of signing the pledge using the pledging coeffi- 
cients estimated based on members of postpolicy pledge 
cohort c, and Pim = F'| ¥9+¥,X; + ¥25,, }, which is the pre- 
dicted probability of a postpolicy student enrolling in col- 
lege based on prepolicy enrollment coefficients. We then 
assess the correspondence between these predicted probabil- 
ities by estimating the following regression for each postpo- 
licy pledge cohort: 


A A A 2 a 3 
PPicm = Ope + ePEim + 7¢PEim +03.PEim (3) 


We estimate Equation 3 as a cubic as we are particularly 
interested in whether there are differential changes in the 
likelihood of signing the pledge along the distribution of 
probabilities of enrolling in college. 


Data for Quantitative Analysis 


The data we utilize are compiled by Washington State’s 
Education Research and Data Center (ERDC). ERDC main- 
tains individual student-level K-12 records for all public 
school students in the state that can be linked to information 
about enrollment in 2- or 4-year colleges in Washington state 
as well as those outside the state (through the National 
Student Clearinghouse). The ERDC data include K-12 stu- 
dent information dating back to the 2005—2006 school year, 
providing us data on two cohorts of students who did not 
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Sign-up rates according to the Washington Student Achievement Council (personal communication, Rachelle Sharpe, 


senior director of student financial aid and support services, Washington Student Achievement Council, August 23, 2013). 


have the opportunity to receive a CBS scholarship (those 
who were in eighth grade in 2005—2006 and 2006-2007) and 
three cohorts who were eligible to sign up (those who were 
in eight grade in 2007-2008, 2008-2009, or 2009-2010). 
We restrict the analysis to students who are known to be 
FRPL eligible, and therefore eligible to sign the pledge, in 
seventh or eighth grade (or eighth or ninth grade for the first 
postpolicy cohort due to a temporary change in eligibility for 
this cohort), which brings our analytical sample size to 
191,337 students.'7 

The CBS program is overseen by the Washington Student 
Achievement Council (WSAC), which “provides strategic 
planning, oversight, and advocacy to support increased stu- 
dent success and higher levels of educational attainment in 
Washington” (WSAC, n.d.). As part of its efforts to support 
the CBS, WSAC monitors sign-up rates across the state and 
by district. WSAC reported substantial success in increasing 
the sign-up rate since the CBS program began. Its initial cal- 
culations, reported in Figure 1, show that the sign-up rate 
was 57% for the first cohort of eligible students and rose to 
85% by the sixth cohort. In 2016, its website stated, 


In 2015, 91% of the Class of 2019 (8th graders whose deadline was 
June 30, 2015) submitted complete applications. This year, 110 
districts had sign-up rates of 92% or higher. Of these, 77 school 
districts saw 100% of their eligible students sign up.'® 


Figure | suggests improvement over time in sign-up rates 
and little room for further improvement in a large number of 
school districts, as these approached 100% participation/ 


sign-up. Our estimates, discussed next, confirm improve- 
ment over time, but as we describe, we found that the base- 
line/first-year sign-up rate to be significantly /ower than the 
figures previously reported by WSAC, suggesting room for 
growth in sign-up rates. 

In particular, our calculations were somewhat different 
than those previously used by WSAC (WSAC has, since the 
release of our working paper, adjusted its method for calculat- 
ing sign-up rates). To calculate the sign-up rate, we utilize 
student-level FRPL data, which capture eligibility in both sev- 
enth and eighth grades corresponding to the policy’s eligibil- 
ity requirements. In Table 1, we show our calculations of the 
sign-up rate for the first three eligible “postpolicy” cohorts. In 
row (A), we find that the number of students signing the 
pledge increased from 14,176 to 18,802 across these three 
cohorts.'” By contrast, WSAC reported (as shown in Figure 1) 
that the number of students signing the pledge increased from 
15,947 to 20,903 across these same three cohorts. Our total 
counts of students who signed the pledge are lower because 
we do not count pledges from foreign exchange students, stu- 
dents in foster care in seventh or eighth grade, students who 
have irreconcilable birthdays across observations, or students 
who did not attend eighth grade in a Washington school. (Note 
that these same students are not included in our denominator.) 
Also, we do not count pledges for which there was no corre- 
sponding student in our K-12 database. 

Next, in row (B), we show that the number of clearly eli- 
gible students remained stable, ranging from 37,519 to 
38,659 across the first three cohorts. This denominator is 


TABLE 1 
Sign-Up Rates 


Expected high school graduation year 


Prepolicy Postpolicy 
2010 2011 2012 2013 2014 
Seventh grade Seventh grade Eighth gradein Seventh grade Seventh grade 


in 2004-2005 or 


in 2005-2006 or 


2007-2008 or in 2007-2008 or in 2008-2009 or 


eighth grade in eighth gradein ninthgradein eighth gradein eighth grade in 
2005-2006 2006-2007 2008-2009 2008-2009 2009-2010 
(A) Number who signed the pledge 25 234 14,176 15,143 18,802 
(B) Number whom we identify as “clearly 39,218 37,450 38,659 37,519 38,491 
eligible” 
(C) Number who signed the pledge and 24 216 13,037 13,943 17,524 
whom we identify as “clearly eligible’ 
Sign-Up Rate 1: (A)/(B) 0.1% 0.6% 36.7% 40.4% 48.9% 
Sign-Up Rate 2: (C)/(B) 0.1% 0.6% 33.7% 37.2% 45.5% 
Sign-Up Rate 2 for Puget Sound counties 0.1% 0.6% 31.0% 34.5% 45.4% 
Sign-Up Rate 2 for other western 0.1% 0.3% 28.1% 32.7% 42.1% 
Washington counties 
Sign-Up Rate 2 for eastern Washington 0.0% 0.6% 41.7% 44.5% 48.4% 


“Clearly eligible” defined as a student that was marked as eligible for free and reduced lunch program at any point during seventh- or eighth-grade year 


(eighth or ninth grade for the first postpolicy cohort). 


substantially larger than the denominator previously reported 
by WSAC (i.e., 28,093 to 29,856 as shown in Figure 1). 
WSAC used the number of FRPL students in seventh grade 
as recorded in October of the corresponding school year. The 
result is an early snapshot of eligibility that does not include 
students who are added to FRPL programs throughout the 
remainder of their seventh grade or at any point during their 
eighth grade.”” 

We believe the sign-up rates we report represent an upper 
bound given that they do not include students who are made 
eligible to sign the pledge by virtue of participation in the 
Supplemental Nutrition Assistance Program, the Food 
Distribution Program on Indian Reservations, or Temporary 
Assistance for Needy Families or who sign the pledge 
despite not being income eligible.*' The sign-up rate we cal- 
culate by dividing (A) by (B) is 36.7% for the first cohort 
and 48.9% for the third cohort. These figures that are sub- 
stantially lower than the 57% and 70% rates reported by 
WSAC in Figure 1, although the increase in our calculated 
rate over time is very close to that reported by WSAC (12.2 
percentage points from first to third cohort vs. WSAC’s 
report of a 13-percentage-point change). 

In an effort to better track progress on the CBS sign-up 
rates, soon after the release of our working paper on July 28, 
2016, WSAC changed the way it calculates sign-up rates to 
have its denominators reflect student mobility into CBS eligi- 
bility and called the new rates “reconciled rates” (Shankster, 
Larson, & Kwakye, 2017). The distinction in the way sign-up 
rates are calculated may well be important because school 
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districts and schools until 2017 could have thought that they 
had only a few eligible students who had not signed the CBS 
pledge when in fact there may be many more.” 

In the last rows of Table 1, we show that there were siz- 
able differences in sign-up rates across the three regions of 
the state, particularly in the first cohort, with eastern 
Washington leading and western Washington counties that 
surround the Puget Sound counties trailing. It is interesting 
to note that although eastern Washington had a high sign-up 
rate for the scholarship program compared to western 
Washington counties (41.7% vs. 28.1% for the first scholar- 
ship cohort), both regions have similar college enrollment 
rates (28.2% vs. 27.1% for the first scholarship cohort). This 
heterogeneity in sign-up rates suggests the possibility of 
uneven communication and understanding of the program 
across the state and provides an argument for the qualitative 
research that we carry out. 

Table 2 presents the descriptive statistics for our analytic 
sample, which consists of those students who are clearly eli- 
gible to sign up. Among eligible students, the share that 
enrolled in any college within 4 years after starting ninth 
grade increased across the two prepolicy cohorts (from 23.6% 
to 25.7%) and then increased again for the first postpolicy 
cohort to 27.2%. Eligible students saw small gains in their 
test scores relative to not-clearly-eligible students (e.g. stu- 
dents who qualify based on family income but do not receive 
free or reduced-priced lunch). Finally, note that eligible stu- 
dents tend to live closer to more undergraduate students than 
not-clearly-eligible students (as indicated by the standardized 


TABLE 2 


Descriptive Statistics for Quantitative Analysis Sample 


Pre-imputation mean and standard deviation 


Prepolicy 


Postpolicy 


Number of observations 
in pooled sample with 


nonmissing data 


Seventh grade 


Seventh grade 


Eighth grade in 


Seventh grade 


Seventh grade 


Prepolicy 


in 20042005 or in 2005-2006 or 2007-2008 or in 2007-2008 or in 2008-2009 or = (two Postpolicy 
eighth grade in eighth grade in ninth gradein eighth gradein eighth gradein cohorts (three cohorts 
Variable 2005-2006 2006-2007 2008-2009 2008-2009 2009-2010 pooled) pooled) 
Dependent variable 
Signed the pledge 0.10% 0.60% 33.70% 37.20% 45.50% 76,668 114,669 
Enrolled in college in 4 years of ninth 23.60% 25.70% 27.20% Incomplete 76,668 114,669 
grade 
Student characteristics 
Reading WASL* 0.38 —0.36 0.32 0.32 0.32 66,038 969,223 
(0.96) (0.95) (1.00) (0.98) (0.97) 
Math WASL* -0.4 0.39 0.37 0.39 —0.38 66,102 96,375 
(0.93) (0.94) (0.94) (0.92) (0.90) 
Took WASL out of grade level 45.7% 55.3% 24.7% 22.8% 19.0% 76,668 114,669 
Took a modified version of WASL 8.5% 8.9% 8.3% 9.0% 8.5% 76,668 114,669 
Age in eighth grade 14.4 14.4 14.4 14.4 14.4 76,668 114,669 
(0.80) (0.50) (0.50) (0.50) (0.50) 
Female 48.0% 48.7% 48.4% 48.7% 48.4% 76,668 114,669 
Hispanic 25.6% 28.4% 30.2% 31.9% 31.6% 76,668 114,669 
Non-Hispanic African American TA% 7.0% 6.6% 6.5% 6.1% 76,668 114,669 
Non-Hispanic Asian 5.9% 5.7% 5.7% 5.6% 5.4% 76,668 114,669 
Non-Hispanic Hawaiian or Pacific 0.1% 0.2% 0.2% 0.3% 0.3% 76,668 114,669 
Islander 
Non-Hispanic Native American or 3.2% 2.3% 2.0% 2.0% 1.8% 76,668 114,669 
Alaskan 
Non-Hispanic and more than one race 7.8% 10.1% 10.3% 10.4% 10.8% 76,668 114,669 
Disability 20.3% 20.4% 21.6% 23.0% 24.1% 76,668 114,669 
Migrant 7.3% 7.9% 8.4% 9.0% 8.6% 76,668 114,669 
Homeless 8.0% 10.3% 11.5% 12.6% 13.6% 76,668 114,669 
Gifted (“highly capable’’) 2.5% 3.9% 3.9% 6.1% 6.2% 76,668 114,669 
Receives bilingual services 13.5% 16.0% 18.5% 21.7% 23.2% 76,668 114,669 
Language spoken at home not English 22.9% 25.8% 28.1% 30.5% 31.3% 76,668 114,669 
Attends public school part-time 0.3% 0.3% 0.5% 0.4% 0.6% 76,668 114,669 
Attended eighth grade in Puget Sound 48.7% 50.1% 49.6% 49.4% 49.3% 76,668 114,669 
county 
Attended eighth grade in western 19.6% 19.1% 19.5% 19.9% 20.3% 76,668 114,669 
Washington county 
Attended eighth grade in eastern 31.7% 30.8% 30.9% 30.7% 30.5% 76,668 114,669 
Washington 
Middle school characteristics 
Average Math WASL* —0.24 —0.20 0.21 0.19 0.22 76,651 114,650 
(0.20) (0.17) (0.17) (0.16) (0.16) 
Eighth-grade fall enrollment 243 238 236 235 233 74,916 113,251 
(106) (101) (106) (106) (105) 
Percentage of student body on FRPL 47.7 47.6 48.5 50.6 51.8 75,874 114,247 
(21.7) (21.4) (21.7) (20.8) (21.1) 
Undergraduate enrollment in a 50-mile 0.10 0.16 0.15 0.15 0.15 76,439 114,438 
radius (1.01) (1.01) (1.00) (1.00) (1.00) 
(standardized across schools within 
cohorts) 
Has a guidance counselor 90.9% 92.1% 93.6% 94.0% 92.7% 76,668 114,669 


Note. Standard deviations are in parentheses. Students are eligible to sign the pledge in eighth or ninth grade for Cohort | and in seventh or eighth grade for subsequent cohorts. 
WASL scores are based on sixth-grade administration for postpolicy cohorts and seventh grade for prepolicy cohorts. WASL = Washington Assessment of Student Learning; FRPL 


= free or reduced-price lunch. 


“Scores are standardized within test type, test grade, and cohort. 
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number of undergraduates within 50 miles being greater than 
zero for eligible students). See Appendix B online for a dis- 
cussion of how missing data are handled. 


Qualitative Data and Analytic Methods 


The goal of our qualitative analysis is to complement the 
quantitative research on uptake of the CBS by providing a 
deeper understanding of how school-level administrators 
understand the CBS, how information is conveyed to stu- 
dents, and what efforts schools are making to get students to 
sign the pledge. As the empirical evidence in Hurwitz and 
Howell (2014) suggests, school administrators, and guid- 
ance counselors in particular, play a vital role in laying the 
groundwork for students to be on a college-ready track and 
successfully enroll in college. We believe such qualitative 
information is key to understanding the CBS take-up rate. 

The qualitative analysis approach was intended to align 
with the structure of the CBS; that is, a set of regional offi- 
cers oversee a geographical area and provide guidance and 
support to school administrators. School administrators then 
(almost always) delegate the program administration to their 
guidance counselor or administrative staff. Interviews were 
conducted at all three levels to see if there was a disconnect 
that was limiting program implementation or that could help 
explain any potential differences between the qualitative and 
quantitative findings. 

Our qualitative analyses draws upon data from semistruc- 
tured interviews. The first set of interviews included the five 
CBS regional officers and their director on the College 
Success Foundation staff. The interviews were conducted 
either individually or in a small group between November 
and December 2015. Interview protocols, which are included 
in Appendix C online, were designed to ensure consistent 
data collection on critical themes across respondents and to 
facilitate systematic analysis. At the same time, they allowed 
for unanticipated themes to emerge. A senior researcher led 
interviews while an analyst took notes. Interviews lasted 
between 60 and 120 min. 

These interviews touched on a variety of topics, 
including 


e Efforts to inform and engage various stakeholders in 
the CBS program (schools, students, parents, commu- 
nity organizations), 

Practices to encourage student uptake, 
Factors limiting student uptake, 

e Evidence that the program is affecting students’ 
behavior, and 

e Factors limiting the program’s success. 


After the interviews, the research team distilled notes 
into analytic themes. We then identified areas of agreement, 
areas of disagreement, and patterns with respect to these 


themes. Overall, these interviews provided a high-level per- 
spective on the CBS’s sign-up activities around the state. 
These interviews were used to inform development of the 
protocol for the second set of interviews with middle school 
principals and guidance counselors. 

For a second set of interviews, we selected a diverse sam- 
ple of middle schools that were based on geographic region 
and the extent of their correspondence (or divergence) in the 
school’s actual sign-up rate to their predicted sign-up rates. 
We predicted each school’s sign-up rate for 2011-2012 as 
follows: 


Signed y, =F (By +B: Xn + BS), (4) 


where Signed, is the school’s predicted sign-up rate, Y,, 
indicates the average of the school’s students’ characteristics 
and S,, are the school’s characteristics in 2011-2012, and By 
6, , and B, are estimated coefficients from Equation 1b, 
based on data from students in the first three CBS-eligible 
cohorts.”* Table 3 shows the distribution of the 27 schools in 
which we conducted interviews. We conducted 30 principal 
and guidance counselor interviews between March and 
August 2016, including 25 guidance counselors (or other 
CBS program leaders) and five principals. We also inter- 
viewed one college readiness coordinator identified through 
professional contacts. 

Analysis of principal and guidance counselor interviews 
proceeded in three steps. First, the research team developed 
the infrastructure of a database corresponding to the inter- 
view protocols, with items organized around the research 
questions and key themes that emerged from the regional 
officer interviews. Second, a trained analyst coded interview 
notes as they were generated, entering information into the 
database with a senior adviser conducting periodic checks 
for quality and consistency of the data. Finally, researchers 
extracted and analyzed data across cases on the key ques- 
tions and themes. Analytic themes that emerged from these 
interviews are discussed in detail in the Results section and 
the Discussion and Conclusion section. 


Results 


Findings for Research Question 1: What Student and 
Middle School Characteristics Are Associated With Signing 
the CBS Pledge? 


Table 4 reports three model specifications for the likeli- 
hood of students signing the pledge. Column (1) of Table 4 
presents raw (not regression-adjusted) differences in sign-up 
rates for each student characteristic, column (2) shows the 
school fixed-effects specification (Equation la), and column 
(3) shows the specification with school characteristics 
(Equation 1b). Although there are some differences in the 
estimated coefficients between the models that do not condi- 
tion on other characteristics (column [1]) and those that do 


TABLE 3 
Respondent Sample by Region and Predicted Versus Actual Sign-Up Rate 


Puget Remainder of western Eastern 
School characterization Sound Washington Washington Total 
Predictably low sign-up rate 1 3 2 6 
Surprisingly low sign-up rate 2 2 2 6 
Typical school 2 2 2 6 
Surprisingly high sign-up rate 2 1 0 3 
Predictably high sign-up rate 2 2 2 6 
Total 9 10 8 27 


Note. Predictably-low-sign-up schools are in the bottom 20% for both actual and predicted sign-up rates. Surprisingly-low-sign-up schools are in the bottom 
20% for actual sign-up rate and in the top 20% for predicted sign-up rate. Typical schools are in the middle quintile for both actual and predicted sign-up 
rate. Surprisingly-high-sign-up schools are in the top 20% for actual sign-up rate and in the bottom 20% for predicted sign-up rate. Predictably-high-sign-up 
schools are in the top 20% for both actual and predicted sign-up rates. 


TABLE 4 
Marginal Effects of Student and School Characteristics on Signing the Pledge 
(1) (2) (3) 
Raw differences not Logit model with Mixed effects logit 
conditioned on other middle school fixed model with school 
Variable characteristics effects characteristics 
Student characteristics 
Math WASL (standardized) .049 (.004)*** 045 (.003)*** .044 (.003)*** 
Reading WASL (standardized) .050 (.004)*** .029 (.003)*** .027 (.003)*** 
Female .073 (.004)*** .057 (.003)*** .056 (.003)*** 
Hispanic .089 (.015)*** .016 (.005)*** .016 (.005)*** 
Non-Hispanic African American .062 (.012)*** .080 (.007)*** .081 (.008)*** 
Non-Hispanic Asian .098 (.013)*** .050 (.008)*** .054 (.005)*** 
Non-Hispanic Hawaiian or Pacific Islander —.102 (.029)*** —.070 (.027)*** —.073 (.024)*** 
Non-Hispanic Native American or Alaskan —.120 (.019)*** —.060 (.014)*** —.057 (.012)*** 
Non-Hispanic and more than one race —.019 (.009)** .022 (.005)*** .020 (.005)*** 
Disability —.091 (.005)*** —.016 (.005)*** —.021 (.004)*** 
Gifted (“highly capable”) 129 (.015)*** 082 (.009)*** .077 (.009)*** 
Migrant 149 (.023)*** .029 (.008)*** .031 (.008)*** 
Receives bilingual services 115 (.013)*** .036 (.006)*** .028 (.006)*** 
Attended eighth grade in remainder of western Washington —.055 (.020)*** —.026 (.020) 
Attended eighth grade in eastern Washington .087 (.023)*** .060 (.023)** 
Middle school characteristics 
Average math WASL (standardized) —.001 (.021) 
Percentage of student body on FRPL .001 (.000)*** 
Undergraduate enrollment in a 50-mile radius (standardized) .040 (.016)** 
Has a guidance counselor .029 (.020) 
McFadden’s pseudo R? 116 .046 
Number of observations 114,193 114,669 


Note. Standard errors are in parentheses. WASL scores are based on sixth-grade administration for postpolicy cohorts and seventh grade for prepolicy 
cohorts. “Raw differences” are computed by an ordinary least squares regression of the outcome on the student characteristic (with no other controls). All 
models have standard errors clustered at the middle school level. Results for taking a modified version of the WASL, taking an out-of-grade-level test, age in 
eighth grade, being homeless, speaking a language other than English at home, attending public school part-time, and middle school eighth-grade enrollment 
are available upon request. WASL = Washington Assessment of Student Learning; FRPL = free or reduced-price lunch. 

*p <.10. **p < .05. ***p < .01 (two tailed). 
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(columns [2] and [3]), there is a fairly consistent pattern of 
the coefficients decreasing in magnitude but maintaining 
their direction and significance with the addition of statistical 
controls. Those coefficients that do change substantially— 
such as taking a modified version of the WASL and attending 
public school part-time—are for smaller student subgroups. 

Some of the groups that are traditionally considered to be 
“advantaged” in terms of educational attainment are more 
likely to sign the pledge. For example, as shown in column 
(1), we find that females, Asian Americans, and students cat- 
egorized as gifted (“highly capable” in the state’s terms) are 
7.3, 9.8, and 12.9 percentage points more likely to sign up, 
respectively, than males, non-Hispanic Whites, and students 
not categorized as gifted. Similarly, we find that a one-stan- 
dard-deviation increase in a student’s math or reading test 
score is associated with a 5-percentage-point increased prob- 
ability of signing the pledge. 

There are, however, some notable exceptions to the pat- 
tern of traditionally advantaged groups being most likely to 
sign up. For instance, as Table 4 shows, we also find that 
Hispanics, non-Hispanic African Americans, migrants, and 
transitional bilingual students are more likely to sign the 
pledge.” Similarly, the regional findings are somewhat sur- 
prising. As Table 4 shows, students who completed eighth 
grade in predominantly rural eastern Washington are more 
likely to sign up than their peers who completed eighth grade 
in the more urban and suburban Puget Sound region. 

Column (2) of Table 4 shows the mean marginal effects 
corresponding to the parameters estimated in Equation la. 
Similar to column (1), we find that a one-standard-deviation 
increase in a student’s math test score is associated with a 
4.5-percentage-point increase in the probability of signing 
the pledge, holding constant all other characteristics. Female 
students are 5.7 percentage points more likely to sign the 
pledge than otherwise comparable males. Relative to non- 
Hispanic White students, Hispanics and non-Hispanic 
African Americans, Asians, and multiracial youth are more 
likely to sign up, whereas Native American or Alaskan 
Native youth are less likely to sign the pledge. Net of these 
student characteristics, the middle school attended clearly 
affects the propensity to sign up; the null hypothesis that the 
addition of the school fixed effects is jointly zero is easily 
rejected (the F statistic is 17). 

In column (3) of Table 4, we present the results including 
middle school characteristics rather than middle school fixed 
effects, corresponding to Equation 1b. We find higher sign- 
up likelihoods in schools that have more undergraduates 
within 50 miles and in schools with more students who are 
FRPL eligible, controlling for the student’s own characteris- 
tics. Interestingly, we find the indicator for a school having a 
guidance counselor is positive, but not statistically signifi- 
cant. Not surprisingly, the specification with middle school 
fixed effects (column [2]) has a substantially stronger fit than 
the model with five school characteristics (column [3]).”° 
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Findings for Research Question 2: How Closely Do the 
Characteristics That Predict Sign-Ups Parallel Pre-CBS 
Program Predictors of College Enrollment? 


Table 5 presents a parallel analysis for the college enroll- 
ment of the prepolicy cohorts, which allows us to examine 
the extent to which the factors predicting CBS sign-ups mir- 
ror factors predicting college enrollment prior to the pro- 
gram’s introduction. Similar to what we found for predicting 
CBS sign-ups, we observe a general pattern of decreasing 
magnitude of the coefficients from the model that is not con- 
ditioned (column [1]) to those with either middle school 
fixed effects or characteristics (columns [2] and [3], respec- 
tively), with little difference between the latter two specifi- 
cations. In the first column, we find that the disparities 
among CBS-eligible students mirror those typically found 
when examining all students (e.g., females, Asian Americans, 
and gifted students are substantially more likely to enroll in 
college “on time’). When estimating the logit specifications, 
we find a remarkable correspondence between the factors 
associated with signing the pledge and enrolling in college. 
The correlation between the marginal effects in the second 
columns of Tables 4 and 5 (i.e., the models with middle 
school fixed effects) is 0.75, and the correlation between the 
marginal effects shown in the third columns of Tables 4 and 
5 (i.e., the models with middle school characteristics) is 
0.69. These significant correlation coefficients clearly sup- 
port the first hypothesis (see the Conceptual Model/Theory 
of Action section) that the factors related to the propensity to 
go to college also predict the likelihood of signing up for the 
CBS program. 

The estimated regression lines based on Equation 3, 
which models the extent to which the predictions generated 
from the sign-up models (Equation 2) are predicted by the 
predictions generated by using the prepolicy college-going 
regressions combined with the postpolicy student character- 
istics, are shown in Figure 2. The figure also shows scatter- 
plots of probabilities of pledging and enrolling in college, 
using a 1 percent random sample of students. Black open 
circles indicate students in Cohort 1, green solid circles indi- 
cate Cohort 2, and red Xs are for Cohort 3. The intercept is 
highest for Cohort 3, reflecting the general increase in the 
likelihood of signing the pledge. For all three cohorts, the 
regression lines have a positive slope indicating that those 
more likely to enroll in college are more likely to sign the 
pledge. The predicted probabilities generated by using the 
prepolicy college-going coefficients are statistically signifi- 
cant predictors of sign-up probabilities, again supporting 
Hypothesis | more formally. 

Moreover, the estimated regression lines in the figure are 
approximately parallel, suggesting little change in this 
underlying relationship, which argues against Hypothesis 2. 
Specifically, we might expect a change in the relationship 
between the factors predicting college going and signing up 
as the CBS program matures and changes college-going 


TABLE 5 


Marginal Effects of Student and School Characteristics on Enrolling in College 


Sign Me Up 


Variable 


(1) (2) (3) 
Raw differences not Logit model with Mixed effects logit 
conditioned on other — middle school fixed model with school 
characteristics effects characteristics 


Student characteristics 

Math WASL (standardized) 

Reading WASL (standardized) 

Female 

Hispanic 
Non-Hispanic African American 
Non-Hispanic Asian 
Non-Hispanic Hawaiian or Pacific Islander 
Non-Hispanic Native American or Alaskan 
Non-Hispanic and more than one race 
Disability 
Gifted (“highly capable’) 
Migrant 


Receives bilingual services 
Attended eighth grade in remainder of western WA 
Attended eighth grade in eastern Washington 
Middle school characteristics 
Average math WASL (standardized) 
Percentage of student body on FRPL 
Undergraduate enrollment in a 50-mile radius (standardized) 
Has a guidance counselor 
McFadden’s pseudo R* 
Number of observations 


157 (.003)*** 
146 (.003)*** 
092 (.004)*## 
—.032 (.008)*** 


092 (.004)*** 
068 (.003)*** 
059 (.003)*** 
—.030 (.006)*** 


092 (.004)*#** 
068 (.003)*** 
059 (.003)*** 
—.028 (.005)*** 


—.006 (.008) 061 (.007)*** 064 (.008)*** 
225 (.014)*** 123 (.008)*** 141 (.010)*** 
—.140 (.044)**# —.076 (.063) ~.078 (.048) 
—.104 (.012)*** —.069 (.014)*** —.065 (.012)*** 
—.064 (.007)*** —.030 (.007)*** —.030 (.006)*** 
—.169 (.004)*** ~.072 (.006)*** —.067 (.005)*** 
335 (.014)*** 165 (.010)*** 190 (.012)*** 
—.005 (.013) 006 (.008) 009 (.008) 
—.031 (.007)*** —.011 (.006)* —.011 (.006)* 
~.004 (.009) 046 (.014)*** 
021 (.011)** 070 (.014)*** 
071 (.025)**# 
—.001 (.000)*** 
—.021 (.010)** 
036 (.012)*** 
146 AN? 
75,785 76,668 


Note. Standard errors are in parentheses. WASL scores are based on sixth-grade administration for postpolicy cohorts and seventh grade for prepolicy 
cohorts. “Raw differences” are computed by an ordinary least squares regression of the outcome on the student characteristic (with no other controls). All 
models have standard errors clustered at the middle school level. Models are estimated on prepolicy cohorts, and the outcome for enrolling in college consid- 
ers on-time college enrollment at any college. Results for taking a modified version of the WASL, taking an out-of-grade-level test, age in eighth grade, being 
homeless, speaking a language other than English at home, attending public school part-time, and middle school eighth-grade enrollment are available upon 
request. WASL = Washington Assessment of Student Learning; FRPL = free or reduced-price lunch. 


*p <.10. **p < .05. ***p < .01 (two tailed). 


expectations (i.e., we should see a change in the relationship 
across eligible cohorts). To more formally test Hypothesis 2, 
we estimate a variant of Equation 3 next, which is a linear 
model that is fully interacted by cohort. In the equation, C, 
and C; are indicators for students in Cohorts 2 and 3. 


= ea 
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This model enables us to test linear interactions between 
cohorts and the predicted probability of a post-policy student 
enrolling in college based on pre-policy enrollment coeffi- 
cients, P,,,,. These interactions are represented by the coef- 
ficients o,-, and o,;. Both interactions are significant at the 
.01 level. However, the significant interactions are direction- 
ally opposite with the interaction term for Cohort 2 being 


—.015 and for Cohort 3 being .056. This inconsistent rela- 
tionship is suggestive of little overall change in the relation- 
ship between signing the pledge and enrolling in college. 
Thus, as was suggested by Figure 2, these findings conflict 
with the second hypothesis, as the relationship between the 
factors predicting college going and signing up for the pro- 
gram are even stronger for the third cohort of eligible stu- 
dents than they are for the first cohort of students. 


Findings for Research Question 3: What Do Program 
Administrators Report Doing to Encourage Student Uptake 
in the Program? 


With respect to school-level sign-up rates, Figure 3 shows 
the relationship between a school’s actual sign-up rate and its 
predicted sign-up rate for the 605 schools in Washington state 
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FIGURE 2. Relationship between probability of enrolling in 
college and probability of signing the pledge. 


Puget Sound Region 


1 


2 
3 

Ze 

a 

z © 

B 

A + 

3 

ga 

s 

< o 

Predicted Sign-up Rate 
Remainder of Western Washington 

2 

a 

a 

J 

q 

a4 

yn 

3 

Fe 

3 

< 

0 2 4 6 
Predicted Sign-up Rate 
Eastern Washington 

e 7- 

a 2 

a 

z <2 

D 

at 

Boa 

8 

a 


0 


Predicted Sign-up Rate 


Marker size is proportional to number of CBS-eligible students 


FIGURE 3. Correspondence of schools’ predicted and actual 
sign-up rates, by region. 


that have program-eligible eighth-grade students. A few 
things merit note here. First, although the actual and pre- 
dicted rates are correlated, there is substantial deviation (the 
raw correlation is .40). This result is consistent with our find- 
ing earlier that school fixed effects were significant predic- 
tors, controlling for student characteristics. Our qualitative 
analysis allows us to explore explanations for this variation. 
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Given the improvement in sign-up rates demonstrated in 
Figure 2 and the variation among schools in expected versus 
actual sign-up rates documented in Figure 3, we examine 
interview data to determine what activities College Bound 
administrators believe are most important for the program’s 
success. Regional officers reported that schools’ success in 
signing up students depends upon (a) district-level “buy-in” 
and support for the program, coupled with a school-level 
champion who takes “ownership” of the program; (b) coun- 
selors or other school staff having access to FRPL data in 
order to target individual students who are eligible for the 
program; (c) guidance counseling staff that are not overbur- 
dened with other responsibilities and have good relation- 
ships with students; and (d) being in a community or district 
with a strong college-going culture. 

All schools included in the qualitative study reported 
using the fall FRPL list to identify eligible students, but 
more than two thirds of guidance counselors interviewed 
reported using additional means to identify eligible students 
who were not on the list. Most prominent among these other 
tactics were sending forms to all students for families to self- 
identify (40%), giving applications to any new student, and 
teachers identifying potential students (20% each). 

Guidance counselors reported using a variety of 
approaches to promote the program and encourage sign-ups. 
The most common activities reported by respondents 
included individual meetings with students (80%) and par- 
ent-teacher conferences (60%); less common were phone 
calls home (40%), other group approaches (32%), other indi- 
vidual approaches (24%) and classroom approaches (20%); 
the tactics least commonly reported included schoolwide 
events (12%) and placing scholarship forms in orientation 
packets (8%) or FRPL application packets (4%).7’ 
Considering students at their own schools, a principal and a 
guidance counselor suggested that those likely to sign the 
pledge were those who are already “looking beyond high 
school” and “excited about college.” 

Guidance counselors have an interest in signing up as 
many students as possible to ensure they do not miss any 
eligible students, and there is no significant consequences 
associated with signing up ineligible students. It is perhaps 
not surprising that fully half of the guidance counselors 
interviewed reported signing students up without checking 
their eligibility. Respondents at more than a third of schools 
reported sharing applications with all students. Two addi- 
tional schools previously sent applications to all families but 
stopped because of complaints by ineligible families or 
because they did not want to promote “false hope.” One 
school asks every student to turn in a form and lets WSAC 
verify the eligibility, whereas at another school, if a parent 
asks about scholarships, the school requests that the parent 
sign the form and sends it to WSAC to verify eligibility. 
Another school signs up every student who ever qualified 
for FRPL. 


Beyond the work of the counselors themselves, another 
important component for CBS program success—identi- 
fied both in the literature and by CBS regional officers—is 
district support for the program. A third of counselor and 
principal respondents could identify a district-level “cham- 
pion,” and almost one fifth reported that their district places 
accountability requirements on schools’ CBS sign-up per- 
formance. Of course, this also shows that two-thirds of 
respondents did not report district accountability require- 
ments related to CBS sign-ups. 

Reflecting on the program’s progress and results, respon- 
dents were not in close agreement about what works, but 
they did largely concur about primary challenges to the CBS 
program’s success. Among the various strategies to support 
the program, respondents most often identified one-on-one 
conversations with parents (42%) or students (35%) as the 
most effective approach. Other strategies considered most 
effective by at least one individual included classroom 
approaches, large incentives, persistence, and having a 
WSAC guest speaker come to the school. The greatest chal- 
lenge to encouraging sign-ups, identified by more than half 
of respondents, was getting signed forms back from stu- 
dents. Other challenges included, for instance, unresponsive 
parents, parents not understanding eligibility requirements, 
the amount of time required for follow-up, and the fact that 
many middle school students are not terribly future oriented, 
so college going is not at the forefront of their minds. For 
example, one counselor, who said the availability of the pro- 
gram could “light kids on fire,” echoed others in noting the 
challenge of promoting college going among middle school- 
ers who often “have no sense of the future.” 

A handful of respondents (13%) stated outright that the 
program had little to no impact on the culture at their school, 
but most regional officers considered the CBS program to be 
a success. Yet they also noted areas where it could be 
improved. In particular, to improve and maintain high sign- 
up rates, respondents pointed to two opportunities. First, 
because district leadership and school champions play such 
important roles, turnover among these positions is a concern. 
Regional officers emphasized the need for more systematic 
approaches, whereby all school faculty and staff have a role 
to play in encouraging program participation and sign-up 
rates serve as part of a school’s evaluation and accountabil- 
ity systems. Second, echoing the principals and counselors, 
there was broad agreement that stakeholders need easier 
access to better and more timely data to identify eligible stu- 
dents, target them for signing up, and support them as they 
move toward college enrollment. 

Regional officers emphasized that program sign-up is just 
the first step. They promoted more attention to students’ 
actual uptake of the scholarship and success in college. Yet, 
maintaining students’ and families’ awareness of the pro- 
gram and encouraging them to adequately plan and prepare 
for college present a different set of issues from encouraging 


Sign Me Up 


program sign-up. Similarly, if the goal of the program is col- 
lege completion, students will require additional supports to 
help them enroll and persist in college. 


Discussion and Conclusion 


Early-commitment pledge programs are fairly new, so it 
is not surprising that we know relatively little about program 
effects and almost nothing about students’ decisions to sign 
up. This is an important gap in the literature given that this 
type of program can help students only if they choose to par- 
ticipate. In this paper, we provide the first evidence of the 
factors that predict the likelihood of students signing 
Washington’s CBS pledge. 

We document that while the state has made considerable 
progress in increasing the number of eligible middle school 
students signing the pledge, sign-up rates were far below 
100% in the early years of the program. This finding con- 
flicts with earlier state reports, suggesting near-universal 
sign-up rates in many districts, which were based on a 
snapshot of eligible students rather than all eligible stu- 
dents. The difference between these state reports and the 
proportion of eligible students that we observe to have 
signed up may be important, as regional officers and guid- 
ance counselors working to encourage both CBS sign-ups 
and college going might have allocated their efforts differ- 
ently if they received more accurate information about 
sign-up rates. 

That said, although all schools report using FRPL lists to 
identify eligible students, more than two thirds of the middle 
school counselors interviewed also report using additional 
means of identifying eligible students. This suggests that 
counselors and other program administrators recognized that 
the FRPL list does not capture all eligible students; in fact, 
about a fifth of counselors believed that calculated rates 
were overestimates of program uptake. When asked about 
the characteristics of students likely to sign up for the CBS, 
the most common responses offered by principals and guid- 
ance counselors were academic strength, a college orienta- 
tion or expectations, and engaged parents.”*® 

The largest process breakdown identified by program 
administrators (counselors and principals) was the difficulty 
in getting signed forms back from students. To account for 
this, counselors often attempted to engage parents directly 
rather than via the student. The most common approach to 
engaging parents was via school conferences (70%). Less 
common approaches include phone calls (44%), letters home 
(33%), and including scholarship forms in school orientation 
or FRPL packets (11%). Nevertheless, the most common 
approach, conferences, suffers from the fact that not all par- 
ents attend conferences and that conferences typically cover 
many topics. In this regard, students represent a linchpin 
through which information, communications, and program 
application paperwork flow. 
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The quantitative findings are largely in line with inter- 
view reports about what drives students to sign up for the 
College Bound program although not without a few excep- 
tions. Regional officers emphasized the importance of 
school and district leadership, resources, and climate in sup- 
porting sign-ups. Specifically, regional officers agreed that 
schools are the most critical partner in targeting eligible stu- 
dents, where they highlighted the need for a “champion” to 
take ownership of the program at the school level and take 
responsibility for its success there. In addition to the school 
champion, respondents noted the importance of a strong 
community or school college-going culture, which may be 
reflected in the positive and significant coefficient for nearby 
undergraduate enrollment shown in Table 4. 

Somewhat surprising is the lack of statistical significance 
of guidance counselors in predicting sign-ups, as regional 
officers had pointed to these individuals as critical to the 
program’s success, regardless of whether they served as the 
champion. It is worth noting, however, that these respon- 
dents did not consider the mere presence of a counselor to be 
adequate. Rather, they noted that heavy workloads can leave 
some counselors struggling to prioritize the program. 
Similarly, they described variability across counselors in 
their ability to develop meaningful relationships with stu- 
dents, which they viewed as key to encouraging students to 
sign up. Finally, they felt that counselors’ access to accurate 
and timely FRPL data was necessary for success but noted 
that many did not have such access, a problem echoed by the 
counselors themselves. 

Our quantitative findings show a positive correlation 
between a school’s predicted sign-up rate and the school’s 
actual sign-up rates, but the correlation, .40, is not over- 
whelmingly high. This, combined with the fact that schools 
have lower sign-up rates than were reflected in public 
reports, suggests it may be beneficial to target resources 
toward encouraging student sign-ups at schools with sur- 
prisingly low sign-up rates. In doing so, we may consider 
the tactics suggested by program administrators, who agree 
that individualized approaches are critical for signing up 
more students and that good data are necessary for such 
implementation. 

Consistent with our first hypothesis and the qualitative 
findings about having a “college orientation,” the individ- 
ual student characteristics that are associated with signing 
the pledge are closely aligned with the characteristics that 
predict whether low-income students matriculate to 4-year 
colleges. High-achieving eligible students, for instance, are 
both more likely to sign the pledge and more likely to go to 
college. 

In contrast to our second hypothesis, the relationship 
between the factors predicting college enrollment and 
signing the pledge did not appreciably change over the 
first three cohorts of program administration. Significantly 
more students are signing up for the CBS program over 
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time, but the fact that the underlying relationship between 
the factors predicting enrollment and pledge signing are 
similar for the early eligible cohorts calls into question 
whether the pledge program is changing college expecta- 
tions significantly for those students with a predictably low 
propensity to go to college. 

The fact that sign-ups are not near to universal is actually 
somewhat surprising given the low cost to sign up and 
potentially high financial reward for doing so. One possibil- 
ity is that the school outreach is too narrowly targeted to 
students. Middle school students unquestionably play a cen- 
tral role in the sign-up decision. However, given that pro- 
gram administrators identified getting signed forms back 
from students as the largest process breakdown and that par- 
ents/families clearly play a role in shaping students’ college 
expectations (Kirk, Lewis-Moss, Nilsen, & Colvin, 2011), 
focusing on family dynamics may prove fruitful. And 
although some guidance counselors report targeting parents/ 
guardians, this is not an explicit part of the CBS program; 
encouraging a parent/family outreach component to the pro- 
gram could be considered as a means to reach those students 
who are least likely now to go to college. 

That said, we believe it is premature to jump to strong 
conclusions about the efficacy of early-commitment pro- 
grams relative to other approaches to increase college access, 
given that the ultimate test of the success of the program is 
whether it is encouraging disadvantaged students to enroll 
and graduate from college. This is the subject of ongoing 
work, 
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Notes 


1. A previous study performed by the BERC Group for the Gates 
Foundation explored the stratification in college readiness, enroll- 
ment, and persistence among pledged College Bound Scholarship 
(CBS) students in the first scholarship cohort (Baker et al., 2013). 
It did not, however, touch on sign-up rates or sign-up rate variation 
that occurs across the state. 

2. The Wisconsin Covenant is not included as eligibility for 
it is not restricted to economically disadvantaged students and 
it was discontinued in 2015. Colorado’s CollegeInvest Early 
Achievers Scholarship is not included as the program was closed 
in 2010. California’s Early Commitment to College and SOAR 
Virginia are not included in this summary because they are not 
available in all schools in the state and programmatic details vary 
across districts. 


3. Perna and Leigh (2017) in their typology of promise pro- 
grams state that they “have a primary goal of increasing higher edu- 
cation attainment, promise a financial award to eligible students, 
have some ‘place’ requirement and focus on the traditional col- 
lege-age population.” Note that this definition does not include an 
element of early commitment. Programs modeled after Georgia’s 
HOPE programs spread from two to 13 states from 1993 to 2003 
(Dynarski, 2004). 

4. B. Long (2004) notes that 4-year colleges in Georgia captured 
a portion of the scholarship by raising tuition and thus “reduced the 
intended benefit of the scholarship and increased the cost of college 
for nonrecipients” (p. 1045). 

5. Surveys of middle school students show that the vast major- 
ity, 88% according to one survey (National Center for Education 
Statistics, 1996), believe they will one day attend and graduate 
from college. Yet academically prepared, low-income students are 
less likely to apply to college as seniors and are “discouraged by 
the complexity of the process of applying for financial aid and col- 
lege admissions, even if they are qualified and enthusiastic about 
going to college” (Avery & Kane, 2004, p. 356). 

6. St. John et al. (2004), for example, used students who 
attended high-poverty schools but who did not sign the pledge as 
the control group. 

7. In Goldhaber, Long, Gratz, and Rooklyn (2019), we use 
difference-in-differences-in-differences specifications and contrast 
changes in outcomes for students who were eligible in the right 
grades (e.g., seventh or eighth grade for most cohorts) versus those 
who faced similar economic hardship but were nearly eligible in 
the wrong grades (e.g., sixth or ninth grade for most cohorts). In 
preliminary results, we find no statistically significant evidence 
that Washington’s CBS affected the likelihood of high school 
graduation, high school grades (getting more than a 2.0 is nega- 
tive and marginally significant in our most recent models), or the 
likelihood of being in juvenile detention, juvenile rehabilitation, 
or incarceration as a young adult. Yet, point estimates suggest the 
program may have cut in half prepolicy gaps between eligible and 
ineligible youths’ likelihood of being incarcerated as a young adult. 

8. For students who attended more than one school in eighth 
grade, including alternative or secondary programs, we used the 
school attended for the most days. 

9. Again, please see Appendix A online for more details about 
the CBS program and pledging requirements for different cohorts 
in particular. 

10. A sixth grader taking the test given to seventh graders would 
be considered to be taking the test out of his or her grade level. 
Washington Assessment of Student Learning (WASL) scores are 
set to missing for these students and then imputed as discussed 
in Appendix B online. For 2008-2009, observations did not have 
reporting grades, and we assume that the test grade is the same as 
the student’s grade level. The high rates of students taking tests 
out of their grade level (see Table 2) are driven by students’ 10th- 
grade standardized tests used for graduation requirements. These 
tests can be retaken and, in some instances, taken early. The change 
in rates between the pre- and postpolicy CBS eras are driven by 
more students being allowed to take the test early in the prepolicy 
era and by a change to the math portion of the graduation test in 
the postpolicy era that reduced the number of retakes. When we 
remove 10th and ninth grades from the out-of-grade-level indica- 
tor, we observe the following rates: 4.5 percent and 4.9 percent for 


Sign Me Up 


the prepolicy era and 5.0 percent, 5.0 percent, and 5.0 percent in the 
postpolicy era. The modified version of WASL is mostly given to 
students with disabilities. 

11. The “incidental parameters” problem that occurs when 
using fixed effects in a logit model with panel data is not a concern 
for us as the number of students at each middle school is typically 
far larger than the numbers that would yield an incidental param- 
eters problem. In similar situations, Cameron and Trivedi (2005) 
prefer the term cluster effects rather than the more commonly used 
term fixed effects, as fixed effects is more appropriately used in the 
context of panel data containing multiple observations of outcomes 
for a single individual, whereas cluster effect refers to a common 
effect occurring for individuals in a cluster, in this case, the school. 
Nonetheless, we follow convention here in using the more familiar 


fixed effects terminology. 


12. We also used the number of undergraduates within 10 or 25 
miles of the middle school, and these produced qualitatively simi- 
lar results. See M. Long and Kennedy (2015) for more information 
about the effect of college proximity on college enrollment decisions 
as well as information on the spatial distribution of Washington’s 
colleges and how that corresponds to the locations of students. 

13. In other words, by the end of October 30th following their 
expected graduation year from high school. 

14. We use seventh-grade WASL scores in_X, rather than sixth- 
grade scores, due to lack of data on sixth-grade scores for earlier 
cohorts. Likewise, we use mean standardized student scores on the 
seventh-grade mathematics WASL as a school characteristics in S_,. 

15. For example, given that girls are more likely to enroll in col- 
lege than boys, we expect girls to be more likely to sign the pledge. 
If we instead find that boys are more likely to sign the pledge, it 
could indicate that (a) the guidance counselors are doing a good 
job at getting the underrepresented group (boys) to sign the pledge, 
(b) the current advantage for low-income boys signing the pledge 
could be working to offset their disadvantage relative to girls in col- 
lege enrollment, and/or (c) efforts focusing on getting low-income 
girls to sign the pledge could be fruitful given their predisposition 
to attend college. 

16. We utilize specifications 1b and 2b rather than 1a and 2a (the 
models with middle school fixed effects) given changes in middle 
schools that occur between the years in which we are estimating 
enrollment models and the years in which we are estimating pledge 
models. 

17. Note that it is not possible with existing administrative data 
to construct a perfect measure of whether a student is eligible to 
sign up for the CBS in middle school, as these data do not have 
information on students who may be income eligible despite not 
receiving free or reduced-price lunch (FRPL), the Supplemental 
Nutrition Assistance Program (SNAP), the Food Distribution 
Program on Indian Reservations, or Temporary Assistance for 
Needy Families (TANF). Based on an analysis of families included 
in the first three waves of the 2008 Survey of Income and Program 
Participation (SIPP), we find that the share of youths who are eligi- 
ble for the CBS only due to income (i.e., who are not FRPL, SNAP, 
or TANF recipients or in foster care) is only 13.3%. The absence of 
income-only eligible students in our analysis could bias our sign-up 
estimates because these students could have a different responsive- 
ness to the CBS program than the students we correctly identify as 
eligible. (See Appendix B online for our analysis of SIPP youth and 
a discussion surrounding the absence of income-eligible students 
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only.) Nevertheless, because we cannot observe the non-FRPL, 
income-only eligible students, we focus our analyses on the FRPL 
sample that we know to be eligible for the CBS and caution readers 
that our findings may not generalize beyond this sample. 

18. Currently available at https://web.archive.org/web/ 
20160920020459/http://www.wsac.wa.gov:80/college-bound. 

19. Note that a small number of students from the prepolicy 
cohorts signed up. These students may include those who were 
retained such that they became part of a subsequent postpo- 
licy cohort. We define a student’s cohort given the first cohort in 
which he or she is observed—thus retained students are counted as 
belonging to their pre-retention cohort. 

20. Personal communication from Rachelle Sharpe, senior 
director of student financial aid and support services, Washington 
Student Achievement Council, December 3, 2015. 

21. In Table 1 we also report a second definition of the sign- 
up rate, the number who signed the pledge and whom we identify 
as “clearly eligible” divided by the number we identify as being 
“clearly eligible” to participate. This lower sign-up rate ranges 
from 33.7% to 45.5% across the three cohorts. 

22. See, for instance, http://www.wsac.wa.gov/sites/default/ 
files/2015.10.23.CBS.School.Districts.pdf. Schools are able to 
view whether their students have signed the pledge and thus can 
monitor sign-up rates (personal communication, R. Sharpe, March 
7, 2016); we are unsure of the extent to which they do so. 

23. Community organizations have been engaged by the state to 
play an active role in helping increase sign-up rates (Power, 2011). 

24. The most recent year for which we had data to compute the 
school’s actual sign-up rate is 2011-2012. 

25. Note that we need to be cautious in assuming that these 
groups are disadvantaged in college enrollment conditional on 
being eligible for the CBS program. Note that 27.2% of all eligible 
Cohort | students enrolled in any college within 4 years after start- 
ing ninth grade, and these listed subgroups had similar if not higher 
college-going rates: 24.0% for eligible Hispanics, 28.4% for eli- 
gible African Americans, 23.9% for eligible migrant students, and 
24.6% for eligible transitional bilingual students. 

26. We use McFadden’s psuedo R’ as a goodness-of-fit measure, 
and it increases from .046 in a specification that includes school 
characteristics only to .116 when we include school fixed effects. 

27. “Other group approaches” include contacting students 
through college incentive programs already present in the 
school, inviting regional representatives to come talk to stu- 
dents in groups, and having a “sign-up week” in the spring when 
counselors talk to eligible students by grade. “Other individual 
approaches” include offering incentives to turn in the forms 
(such as candy, bracelets, etc.), stopping students in hallways to 
remind them, and setting up tables at family nights and other 
school events. “Other classroom approaches” include making 
presentations in homeroom or other classrooms, asking teachers 
to promote the material, and including scholarship forms with 
field trip permission slips. 

28. Engaged parents are characterized by their level of respon- 
siveness to the interventions employed by program administrators. 
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